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LIO : Entry 1 of 3 



File: USPT 



Sep 1, 1998 



DOCUMENT- IDENTIFIER: US 5802361 A 

TITLE: Method and system for searching graphic images and videos 



Detailed Description Text (69) : 

If there is no express precedence in the search inquiry, the side information files 
115 are evaluated based on statistics built up by the system 100 on the optimal term 
evaluation strategy in the past. Alternatively, the side information files 115 may be 
analyzed based on the computational requirements associated with each type of side 
information file, with less computationally intensive side information files being 
evaluated first. Other considerations may also used to determine the order of 
evaluation of the image attributes and side information files 115 such as evaluating 
color image attributes prior to texture or motion, or evaluating text image attributes 
before all others . These orderings are used to improve performance by evaluating those 
image attributes that have either the greatest immediate semantic value and likeliness 
of accurate reflection of the user's intended meaning for the search query, or that 
require the least amount of computational power to evaluate, thus leaving those 
attributes that take more computational resources for evaluation over the smallest 
potential search spaces. Thus, in the prior example, the color histogram side 
information file would be evaluated first, and all images containing 

color {H [150 .degree ., 245 .degree .] ; S[0.05, 0.70]) would be identified, since color is a 
very strong indicator of a correct match of a "blue sky" {where this is the hue and 
saturation setting for the color of a blue sky) and a user is most likely to 
accurately specify the color of a "blue sky" when constructing the search inquiry. 
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L13 : Entry 1 of 3 



File: USPT 



Jun 2, 1998 



DOCUMENT- IDENTIFIER: US 5761655 A 

TITLE: Image file storage and retrieval system 



US PATENT NO. (1) : 
5761655 

Brief Summary Text (5) : 

The present invention is directed to methods and apparatus for creating and retrieving 
image data files in a computer system and, more particularly, to a system in which 
tens of thousands or even hundreds of thousands of files must be created and searched 
for retrieval, where the file retrieval process requires narrowing the number of files 
presented to the user and manually choosing the desired files out of the set presented 
and where selection can be by many different kinds of similarities thereby providing 
multi-dimensional organization of the files and also allowing the organization of many 
storage devices integrated with media autochangers to automatically bring off line 
files on line for access. 

Brief Summary Text (13) : 

Conventional programs designed specifically for organizing images provide the same 
selection ability but typically save the thumbnails for each newly encountered image 
file in a master database or append the thumbnails to the actual file itself. This 
allows more efficient presentation of images in the future, also organizers will often 
allow keywords to be assigned to files or groups of files. These keywords can then be 
used to select a smaller group (i.e. only those tagged with the given keywords) of 
images from which to select. This is very important when a large number of images are 
being stored. Existing programs using this method of image selection are Curator by 
Solutions International. 

Brief Summary Text (20) : 

It is an additional object of the present invention to allow selection of those image 
files matching m out of n selected keywords including super -keywords allowing for a 
fudge -factor where some relevant images may not be tagged with all selected keywords 
and where super -keywords are actually entire templates of selection criteria which 
have been assigned a keyword name. 

Current US Cross Reference Classification (1) : 
707/3 
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L13 : Entry 2 of 3 



File: USPT 



May 12, 1998 



DOCUMENT- IDENTIFIER: US 57512 86 A 
TITLE : Image query system and method 



US PATENT NO. (1) : 
5751286 

Abstract Text (1) : 

Images in an image database are searched in response to queries which include the 
visual characteristics of the images such as colors, textures, shapes, and sizes, as 
well as by textual tags appended to the images . Queries are constructed in an image 
query construction area in response to values of representations of the visual 
characteristics and to locations of the representations in the image query 
construction area. 

Brief Summary Text (12) : 

A more sophisticated method for retrieving images similar to a given image is given in 
Mikihiro loka. "A Method of Defining the Similarity of Images on the Basis of Color 
Information", Technical Resort RT-0030, IBM Tokyo Research Lab, 1989. Here, each image 
in the database {actually, the sub image of each image containing a single, dominant 
object the image) is partitioned into blocks, for example. 2 5 blocks. Within each 
block, the reduced bucket histogram, h, (say, 256 buckets) is computed. Given a query , 
image or object, is also partitioned into the same number of blocks and the histograms 
computed. A similarity measure s (h. sub. query . sbsb sub . image 

h, sub. database. sbsb. -- .sub. item) is defined on the color histograms computed in the 
blocks, and the measure is extended to images as: 

Brief Summary Text (16) : 

The general problem solved by this invention is that of retrieving images from an 
online image database. Standard methods use image identifiers and/or -text or keywords 
associated with each image. Other approaches, such as those described in the prior 
art, use the colors that appear in the images. 

Detailed Description Text (23) : 

In the first embodiment of the invention, four selection windows (containers) are 
employed: a colpr container 91, a texture container 92, a shape container 93, and a- 
category container 94. The containers 91, 92 and 93 correspond to characteristics 
which are innate in an image and which can be calculated as described above from image 
content. The category characteristic is an arbitrary textual tag which can be appended 
to an image or an image mask. Each of the selection containers 91-94 contains one or 
more icons, which are referred to, preferably, as "thumbnails". Each thumbnail in one 
of the image characteristic containers 91, 92 and 93 corresponds to a prespecified 
value for an image characteristic. For example, the "RED" thumbnail 96 .in the color 
selection container 91 corresponds to a precalculated value for red taking into 
account all the components of a particular display scheme which contribute to the 
particular shade of red given by the thumbnail 96. In the texture selection container 
92, the "GRASS" thumbnail 97 corresponds to a set of precalculated textural values. In 
the shape selection container 93, the "TRIANGLE" thumbnail 98 corresponds to a 
quantized representation of a triangle. 

Current US Cross Reference Classification (5) : 
382/305 
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M. Kurokawa, "An Approach to Retrieving Images by Using their Pictorial Features", IBM 
Research, Japan, Sep. 1989. 
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L15 : Entry 1 of 3 



File: USPT 



Jun 5, 2001 



DOCUMENT- IDENTIFIER: US 624 3 713 Bl 

TITLE: Multimedia document retrieval by application of multimedia queries to a unified 
index of multimedia data for a plurality of multimedia data types 



US PATENT NO. (1) : 
6243713 

Brief Summary Text (20) : 

The various tokens resulting from the component processing are all stored in the 
unified multimedia index, even though they represent different types of multimedia 
data. Thus, text, image, audio, and video, and other tokens are all combined into the 
same index, thereby facilitating robust retrieval, cross -media relevancy ranking, and 
sharing of indexing, search, and retrieval modules. In a preferred embodiment, the 
multimedia index is an inverted index with each token, regardless of its type, 
associated with a set of documents containing the data which the token represents. 
Each of these indexed documents in turn contains a list with the position of each 
occurrence of the token in the document, and the reference data for the occurrence of 
the token. 

Brief Summary Text (22) : 

A preferred embodiment for processing the queries uses a coarse and fine-grain scoring 
of documents, but with the same query. Coarse -grain scoring evaluates the query with 
respect to only whether a document includes or does not include tokens of the query, 
essentially a Boolean analysis. Fine-grain ranking uses proximity operators to compare 
positional information for the tokens, and to compare underlying reference data 
between query tokens and document tokens. 

Detailed Description Text (4) : 

The inputs to the indexing pipeline 192 are multimedia, or compound documents 
containing various types of multimedia components (though of course the present 
invention may be used to index documents that have only text, as well as images, 
videos, or audio recordings by themselves) . A compound document may be structured in 
one of two ways: 1) the multimedia components are specified by reference, or; 2) the 
multimedia components are embedded directly in the data of the document. An example of 
the first structure are HTML documents, which incorporate external files by reference. 
For example, an image may be incorporated with the HTML tag <IMG SRC>, such as in <IMG 
SRC= "http : /abc/ image . gif "> . 

Detailed Description Text (11) : 

From the generated query components, including additional components from query- 
expansion 180, the retrieval pipeline creates 190 a query structure that organizes the 
selected tokens along with search operators into an expression that can be used to 
evaluate or score individual compound documents. This query structure is then 
processed 200 against the multimedia index 140 to select and score a number of 
compound documents that satisfy the query. In a preferred embodiment, the selection of 
documents involves two levels of relevancy scoring. Coarse-grain relevancy scoring is 
used to select an initial set of candidate -documents by evaluating the query structure 
using data indicative of • whether a document contains the query tokens. With the 
smaller candidate set, a fine-grain relevancy scoring uses the reference data of the 
tokens to evaluate both proximity operators, and similarity operators. The best 
scoring documents from the candidate documents form the search results . 



Detailed Description Text (28) 
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As an example of component tags , in an RTF document image components start with 
"{ .backslash. pict" and end with "}". In HTML documents, images start with "<IMG SRC=" 
and end in ">". Searching 504, 506 for component tags may involve recognizing tags 
regardless of embedded spaces or mixed upper/ lower case. 

Detailed Description Text {36} : 

The tokens are then further filtered and normalized 630, for example, eliminating 
tokens for less significant data (e.g. helping verbs, or infrequently appearing color 
tokens) , For each of the remaining tokens, the position of the token within the 
multimedia component or the document itself is determined 640. This information is 
useful to perform. true proj^imity searching of tokens, particularly of tokens for 
different types of multimedia components, and relevancy ranking of documents. In 
addition to relevancy ranking, the document position may also be used for document 
highlighting (i.e. identifying in the retrieval user interface the parts of a 
multimedia document which were matched to the user's query) . 

Detailed Description Text (37) : 

Finally, each token created may have computed 65 0 for it an arbitrary amount of 
"reference data" . Reference data typically identifies the position of the token within 
the multimedia document, plus any additional data required for fine-grain relevancy 
ranking for the object (usually determined by the type of the object) . This additional 
data is again dependent on the type of the multimedia component. 

Detailed Description Text (192) : 

Searching is broken into three major stages: Coarse-grain searching 1402, fine-grain 
searching 14 02, and final scoring 1406. This division is done primarily for 
performance reasons. While it is certainly possible to do fine-grain relevancy ranking 
on all documents in a database, such a method would be very slow. 

Detailed Description Text (193) : 

Coarse-grain searching 1402 is an initial relevancy ranking stage used to identify a 
"rich subset" of documents that likely satisfy the user's query. Coarse -grain 
searching 14 02 generally considers the existence or absence of tokens in a document 
when computing the value of the query structure for a document. Thus, as noted above, 
a query token receives a token value of 1.0 if it is present in the document, and a 
token value of 0.0 if it is absent. The only data required for this computation is the 
list of documents 904 which contain each token. For each document, the query structure 
is computed. This gives the coarse-grain score for each document. The result is a 
candidate document list 1412. 

Detailed Description Text (202) : 

Once the computations are finished, a set 1412 of the best documents are selected 1508 
as the candidate documents to fine-grain searching 1404. The number of candidate 
documents selected can be specified by the user as either an absolute number, or as a 
threshold based on the coarse-grain relevancy ranking score, or other filtering means. 



Detailed Description Text (227) : 

These scores are combined 1520 in a linear equation to produce the final relevancy 
score for the document. The constants of the linear equation can be determined through 
a statistical regression analysis by executing, for example, 50 queries and evaluating 
the top 2 00 documents for each. One preferred embodiment uses a formula such as 
A* (NUM_HITS/NUM_WORDS) +B*MAX_HIT+C*COARSE_RANK, where A, B, and C are adjusted to 
balance the saliency of these different factors, based on analysis of the document 
set. This formula may also be adjusted for small documents, for example to have 
NUM_WORDS set to 100 for documents which have fewer than 100 words, or some other 
constant. Obviously, different scoring equations with different constants may be used. 



Current US Original Classification (1) : 
707/104 . 1 

Current US Cross Reference Classification (3) : 
707/3 
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Other Reference Publication (17) : 

Lu, Z., Callan, J. P., Croft, W.B., "Measures in Collection Ranking Evaluation," 
University of Massachusetts, Technical Report TR96-39, 1996. 



CLAIMS : 



30. A computer implemented method for retrieving documents, including compound 
documents, each of the compound documents including at least two different multimedia 
components, each multimedia component having data of one of a plurality of multimedia 
data types, the method comprising: 

receiving a compound query including at least one non-textual multimedia query 
component ; 

processing the compound query to generate a query structure comprising a set of 
tokens, including at least one token representing a non-textual multimedia data type; 
and 

evaluating the query structure with respect to a multimedia index to selectively 
retrieve compound documents that satisfy the compound query, each retrieved compound 
document including data represented by at least one token generated from the compound 
query. 

42. The method of claim 30, wherein the multimedia index includes position 
information, and evaluating the query structure comprising: 

evaluating the query structure with respect to the multimedia index a first time to 
selectively retrieve candidate documents that satisfy the compound query by 
determining a document score for each of a number of documents based only on a 
presence or absence of tokens from the query structure in the document, and selecting 
a number of best scoring documents as the candidate documents; and 

evaluating the query structure with respect to the multimedia index a second nine by 
computing a final document score of each candidate document as a function of the 
position of each token from the query structure in the candidate document, using the 
position information from the multimedia index. 



44. The method of claim 30, wherein: 



the multimedia index comprises tokens indexed to a list of documents and reference 
data associated with each listed document; and 

wherein processing the compound query comprises: 

determining reference data for at least one of the generated tokens of the query 
structure; and 

including the reference data in the query structure; 'and 
wherein evaluating the query structure comprises: 

evaluating the query structure with respect to the multimedia index a first time to 
selectively retrieve candidate documents that satisfy the compound query by 
determining a document score for each of a number of documents based only on a 
presence or absence of tokens from the query structure in the document, and selecting 
a number of best scoring documents as the candidate documents; and 

evaluating the query structure with respect to the multimedia index a second time by 
computing a final document score of each candidate document based on a comparison of 
the reference data in the query structure with the reference data in the multimedia 
index. 

46. A computer implemented method for retrieving documents, including compound 
documents containing both text and at least one image, each image having a 
predetermined position within the document, the method comprising: 
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receiving a compound query including text and at least one image; 

processing the compound query to generate a query structure including at least one 
token representing the text, at least one token representing the image, and at least 
one search operator defining a logical or proximity relationship between the text and 
the image; and 

evaluating the query structure with respect to a multimedia index to selectively 
retrieve compound documents that satisfy the compound query, the multimedia index 
comprising a plurality of tokens representing texts and images, each of the tokens 
representing text associated with a text string, each of the tokens representing an 
image associated with an image feature and with reference data specifying at least a 
position of the image represented by the token within at least one compound document 
including the image, each token indexed to a list of documents including the data 
represented by the token. 
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L15 : Entry 2 of 3 



File: USPT 



May 12, 1998 



DOCUMENT- IDENTIFIER: US 57512 86 A 
TITLE: Image query system and method 



US PATENT NO. (1) : 
5751286 

Abstract Text (1) : 

Images in an image database are searched in response to queries which include the 
visual characteristics of the images such as colors, textures, shapes, and sizes, as 
well as by textual tags appended to the images . Queries are constructed in an image 
query construction area in response to values of representations of the visual 
characteristics and to locations of the representations in the image query 
construction area. 

Detailed Description Text (23) : 

In the first embodiment of the invention, four selection windows (containers) are 
employed: a color container 91, a texture container 92, a shape container 93, and a 
category container 94. The containers 91, 92 and 93 correspond to characteristics 
which are innate in an image and which can be calculated as described above from image 
content. The category characteristic is an arbitrary textual tag which can be appended 
to an image or an image mask. Each of the selection containers 91-94 contains one or 
more icons, which are referred to, preferably, as "thumbnails". Each thumbnail in one 
of the image characteristic containers 91, 92 and 93 corresponds to a prespecified 
value for an image characteristic. For example, the "RED" thumbnail 96 in the color 
selection container 91 corresponds to a precalculated value for red taking into 
account all the components of a particular display scheme which contribute to the 
particular shade of red given by the thumbnail 96. In the texture selection container 
92, the "GRASS" thumbnail 97 corresponds to a set of precalculated textural values. In 
the shape selection container 93, the "TRIANGLE" thumbnail 98 corresponds to a 
quantized representation of a triangle. 

Detailed Description Text (30) : 

A context dialogue box illustrated in FIG. 7 is displayed when the cursor is placed on 
a thumbnail and a preselected mouse button is clicked. The context dialogue box 
provides two control bars for setting, respectively, distance and weight of the 
thumbnail. Preferably, the setting of the distance control thresholds the value of the 
thumbnail characteristic, indicating how close the value must be to the value of a 
mask characteristic in order to include the characteristic in a query calculation. The 
setting of the weight control determines the weighting factor for this characteristic. 
Manifestly, the adjustment of thumbnail values for multiple thumbnails can establish a 
relative weighting among thumbnail values for query construction. For example, 
suppose, in the "BEAR" and "WATER" example described above, a user is very interested 
in image color but less interested in the texture of images. In this case, the user 
would increase the color attribute, but lighten the weight of texture. 

Detailed Description Text (35) : 

FIGS. lOA and lOB comprise a flow chart describing a process for practicing the first 
embodiment of the invention. In FIG. lOA, the process is initiated in step 210 by 
displaying an example image window such as the window 90 illustrated in FIG. 5. A 
query -by -image content procedure is initiated in step 211 by, for example, selecting 
the query button 89 in the example image window 90, which issues a RUN QUERY command. 
In step 212. selection windows are selected and displayed using the view pulldown 
illustrated in FIG. 8B. Thumbnails are then dragged from selection windows and dropped 



Record Display Form 



http://westl?rs: 8002/b in/gate.exe?f=doc&s. .^^_Message=&p_doccnt= 1 &p_doc_ 1 =PTFK WIC 



in the example image window in step 213. For as long the RUN QUERY command is not 
selected, the negative exit is taken from decision 214 and the sample image being 
constructed in the example image window can be edited or added to in step 213. When 
the RUN QUERY command is selected, the positive exit is taken from decision 214 and, 
for each image in the database, the procedure executes a sequence of steps beginning 
with decision 216 and ending with step 225. Recall that selection of the RUN QUERY 
command sends a set of data structures describing objects in the sample image to the 
QBIC function. Each object data structure is inspected to determine whether the object 
includes a category thumbnail. If so, the positive exit is taken from decision 216 and 
the masks in the current image are filtered by the category code in the data structure 
in step 217. The decision 216 then establishes the set of masks of the current image 
which will be analyzed for composite distance with respect to the image characteristic 
values returned in the data structure for this object. The set of masks established in 
the decision step 216 are then subjected to the composite distance calculations as 
described above with reference to equations (1) - (3) . In step 218, the layout option is 
inspected. If enabled, the composite distance calculations for each mask are 
thresholded by mask/object distance in step 220. In step 222, taking account of the 
outcome of decision 218, composite distances between objects of the sample image and 
the filtered and thresholded masks of the current image are calculated using the 
weight, distance, and thumbnail values return for the object in step 222. Next, in 
step 224, the composite distance for the image is calculated. If the image is not the 
last in the set of images, the negative exit is taken from step 225 and the composite 
distance for the next image is calculated as just described. Otherwise, the positive 
exit is taken in step 225, the images are ranked according to the results of the 
composite distance calculations in step 227 and the result of the ranking are 
displayed in step 228. 

Current US Cross Reference Classification (5) : 
382/305 
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L19: Entry 1 of 1 



File: USPT 



Jun 5, 2001 



DOCUMENT- IDENTIFIER: US 624 3 713 Bl 

TITLE: Multimedia document retrieval by application of multimedia queries to a unified 
index of multimedia data for a plurality of multimedia data types 

US PATENT NO. (1) : 
6243713 



Detailed Description Text (4) : 

The inputs to the indexing pipeline 192 are multimedia, or compound documents 
containing various types of multimedia components (though of course the present 
invention may be used to index documents that have only text, as well as images, 
videos, or audio recordings by themselves) . A compound document may be structured in 
one of two ways: 1} the multimedia components are specified by reference, or; 2) the 
multimedia components are embedded directly in the data of the document. An example of 
the first structure are HTML documents, which incorporate external files by reference . 



For example, an image may be incorporated with the HTML tag rzi 



such as in 



Detailed Description Text (28) : 

As an example of component tags, in an RTF document image components start with 




" { .backslash.pict" and end with "}". In HTML documents, images start with " 
Searching 504, 506 for component tags may involve recognizing tags regardless of 
embedded spaces or mixed upper/lower case. 
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L22: Entry 1 of 1 File: USPT Jul 27, 1999 



DOCUMENT- IDENTIFIER: US 5 93 0783 A 

TITLE: Semantic and cognition based image retrieval 



Abstract Text (1) : 

A computer implemented method for searching and retrieving images contained within a 
database of images in which both semantic and cognitive methodologies are utilized. 
The method accepts a semantic and cognitive description of an' image to be searched 
from a user, and successively refines the search utilizing semantic and cognitive 
methodologies and then ranking the results for presentation to the user. 

Brief Summary Text (2) : 

The present invention relates generally to the field of databases and in particular to 
a method and system for retrieving images contained within a database using both 
semantic expressions and image examples. 

Detailed Description Text (133) : 

Those skilled in the art will quickly appreciate that SEMCOG may support a variety of 
interfaces. One such interface is shown in FIG. 10 where an HTML -based user interface 
is used for the main menu of image queries. Users can use this interface to specify 
CSQL directly. The windows for input sketches and specifying images will pop up by 
pressing familiar buttons or other controls. In a similar manner, FIG. 11 shows the 
interfaces implemented using Java for input sketches. FIG. 12 show the interface for 
input images by specifying the file path. 

Current US Original Classification (1) : 
707/1 

Current US Cross Reference Classification (2) : 
707/104 .1 

Current US Cross Reference Classification (3) : 
707/3 

CLAIMS : 

1 . A method of searching a database of images and retrieving at least one image 
contained in said database of images, comprising the steps of: 

i) generating a first set of images which match a semantic criteria of the at least 
one image to be retrieved; 

ii) generating a second set of images from the first set of images, which match a 
cognition criteria of the at least one image to be retrieved; 

iii) generating a third set of images from the second set of images, which match a 
scene criteria of the at least one image to be retrieved; 

iv) ranking the third set of images based on similarity to the semantic criteria and 
the cognition criteria of the at least one image to be retrieved into a set of 
results; and 
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v) presenting the user with the set of results. 

22. A method of searching a database of images and retrieving at least one image 
contained in said database of images, comprising the steps of: 

i) generating a first set of images which match a semantics criteria of the at least 
one image to be retrieved; 

ii) generating a second set of images from said first set of images which match a 
cognition criteria of the at least one image to be retrieved and; 

iii) providing said at least one image contained in said database of images to a user. 

23. The method of claim 22, further comprising the step of retrieving a third set of 
images from said second set of images which match a scene criteria of the at least one 
image to be retrieved, prior to providing said at least one image to said user. 

27. A method of searching a database of images and retrieving at least one image 
contained in said database of images, comprising the steps of: 

i) generating a set of resulting images matching a semantics-based criteria and a 
cognition-based criteria of said at least one image to be retrieved from said database 
of images; and 

ii) presenting said set of resulting images to a user. 



